Troubleshooting the System


Troubleshooting the System
 
 
This chapter provides information and instructions for using the system command line interface (CLI) for troubleshooting any issues that may arise during system operation.
Refer to the ASR 5000 Installation Guide for comprehensive descriptions of the hardware components addressed by these troubleshooting procedures.
The following topics are included:
Detecting Faulty Hardware
When power is applied to the chassis, power is sequentially applied to management cards, application cards and line cards.
Each PFU, application and line card installed in the system incorporates light emitting diodes (LEDs) that indicate its operating status. This section describes how to use these status LEDs to verify that all of the installed components are functioning properly.
Important: As the system progresses through its boot process, some cards will not exhibit immediate LED activity. Allow several minutes to elapse after a reboot is initiates before checking the LEDs on the various cards to verify that the boot process has successfully completed.
Using the CLI to View Status LEDs
The status of application and line card LEDs can be viewed through the CLI by entering the show leds all command in Exec mode.
The following displays a sample of this command’s output.
Slot 01: Run/Fail: Green | Active: Off | Standby: Green
Slot 08: Run/Fail: Green | Active: Green | Standby: Off
Status: Green | Service: Off |
Slot 09: Run/Fail: Green | Active: Off | Standby: Green
Status: Green | Service: Off |
Slot 12: Run/Fail: Green | Active: Green | Standby: Off
Slot 14: Run/Fail: Green | Active: Green | Standby: Off
Slot 17: Run/Fail: Green | Active: Green | Standby: Off
Slot 24: Run/Fail: Green | Active: Green | Standby: Off
Slot 25: Run/Fail: Green | Active: Off | Standby: Green
Slot 30: Run/Fail: Green | Active: Green | Standby: Off
Slot 33: Run/Fail: Green | Active: Off | Standby: Off
Slot 40: Run/Fail: Green | Active: Green | Standby: Off
The status of the two Power Filter Units (PFUs) can be viewed by entering the show power chassis command in the Exec mode.
Checking the LED on the PFU
Each PFU has a single status LED labeled POWER.
This LED should be green for normal operating conditions.
PFU LED Location
The possible states for this LED are described in the following table. If the LED is not green, use the troubleshooting information below to diagnose the problem.
PFU POWER LED States
 
Checking the LEDs on the SMC
Each SMC is equipped with the following LEDs as shown in the accompanying figure:
 
SMC LEDs
The possible states for all SMC LEDs are described in the sections that follow.
SMC Run/Fail LED States
The SMC Run/Fail LED indicates the overall status of the card. This LED should be green for normal operation.
The possible states for this LED are described in the following table. If the LED is not green, use the troubleshooting information in the table to diagnose the problem.
SMC Run/Fail LED States
The Monitoring the System chapter in this guide for show commands, the outputs of which will assist in further determining the problem.
The Configuring and Viewing System Logs chapter in this guide for information on how to view logs.
The SNMP MIB Reference for information on associated status and alarm conditions.
Verify that the POWER LEDs on the PFUs are green. If they are not, refer to the Checking the LED on the PFU section for troubleshooting information.
SMC Active LED States
The Active LED on the SMC indicates that the software is loaded on the card and it is ready for operation. For the SMC installed in chassis slot 8, this LED should be green for normal operation. For the SMC installed in slot 9, this LED should be off for normal operation.
The possible states for this LED are described in the following table. If the LED is not green, use the troubleshooting information in the table to diagnose the problem.
SMC Active LED States
Verify that the Standby LED on the redundant SMC is also blinking green. If so, there is an issue with the active SMC. Refer to one or more of the following to help analyze this problem:
The Monitoring the System chapter in this guide for show commands, the outputs of which will assist in further determining the problem.
The Configuring and Viewing System Logs chapter in this guide for information on how to view logs.
The SNMP MIB Reference for information on associated status and alarm conditions.
Card is not receiving power. OR Card is in Standby Mode.
Verify that the Run/Fail LED is green. If so, the card is receiving power and POST test results are positive. If it is off, please refer to the SMC Run/Fail LED States section for troubleshooting information.
Check the state of the Standby LED. If it is green, the card is in standby mode. If needed, refer to the Configuring Packet Processing Card and Line Card Availability section of the Configuring System Settings chapter in this guide for information on making the card active.
SMC Standby LED States
The Standby LED on the SMC indicates that software is loaded on the card, but it is serving as a redundant component. For the SMC installed in slot 9, this LED should be green for normal operation. For the SMC installed in slot 8, this LED should be off for normal operation.
The possible states for this LED are described in the following table. If the LED is not green, use the troubleshooting information in the table to diagnose the problem.
SMC Standby LED States
Verify that the Active LED on the redundant SMC is also blinking green. If so, there is an issue with the active SMC. Refer to one or more of the following to help analyze this problem:
The Monitoring the System chapter in this guide for show commands, the outputs of which will assist in further determining the problem.
The Configuring and Viewing System Logs chapter in this guide for information on how to view logs.
The SNMP MIB Reference for information on associated status and alarm conditions.
Card is not receiving power. OR Card is in Active Mode.
Verify that the Run/Fail LED is green. If so, the card is receiving power and POST test results are positive. If it is off, please refer to the SMC Run/Fail LED States section for troubleshooting information.
Check the state of the Active LED. If it is green, the card is in active mode. If needed, refer to the Manually Initiating a Management Card Switchover section for information on configuring the card to serve as a redundant component.
 
SMC Status LED States
The Status LEDs on the SMC indicate the status of system level hardware such as installed cards, fans, and PFUs. This LED is green during normal operation.
The possible states for this LED are described in the following table. If the LED is not green, use the troubleshooting information also provided to diagnose the problem.
SMC Status LED States
Check the Run/Fail LEDs for all installed application cards, and line cards. If any are red or off, refer to the troubleshooting information in this chapter pertaining to that device.
The Monitoring the System chapter in this guide for show commands, the outputs of which will assist in further determining the problem.
The Configuring and Viewing System Logs chapter in this guide for information on how to view logs.
The SNMP MIB Reference for information on associated status and alarm conditions.
Verify that the Run/Fail LED is green. If so, the card is receiving power and POST test results are positive. If it is off, refer to the SMC Run/Fail LED States section for troubleshooting information.
 
SMC Service LED States
The Service LEDs on the SMCs indicate that the system requires maintenance or service (for example, the system could not locate a a valid software image at boot-up, or a high temperature condition exists).
This LED is off during normal operation.
The possible states for this LED are described in the following table. If the LED is not green, use the troubleshooting information in the table to diagnose the problem.
SMC Service LED States
The Monitoring the System chapter in this guide for show commands, the outputs of which will assist in further determining the problem.
The Configuring and Viewing System Logs chapter in this guide for information on how to view logs.
The SNMP MIB Reference for information on associated status and alarm conditions.
 
SMC Busy LED States
The Busy LEDs on the SMCs indicate that there is activity on one of their memory devices. Activity is displayed for the following memory devices:
The possible states for this LED are described in the following table. If the LED is not green, use the troubleshooting information in the table to diagnose the problem.
SMC Busy LED States
NOTE: You should wait until this LED is off before removing the SMC from the chassis. This practice ensures the integrity of all data being transferred to or from the memory device.
 
Checking the LEDs on the Packet Processing Cards
The ASR 5000 supports a variety of packet processing cards (PSCn and PPC). For detailed information about the types of cards and their applications, refer to the ASR 5000 Installation Guide.
Each packet processing card is equipped with the status LEDs listed below:
 
Packet Processing Card LEDs
The possible states for all packet processing card LEDs are described below.
Packet Processing Card Run/Fail LED States
The packet processing card Run/Fail LED indicates the overall status of the card. This LED should be green for normal operation.
The possible states for this LED are described in the following table. If the LED is not green, use the troubleshooting information in the table to diagnose the problem.
Packet Processing Card Run/Fail LED States
The Monitoring the System chapter in this guide for show commands, the outputs of which will assist in further determining the problem.
The Configuring and Viewing System Logs chapter in this guide for information on how to view logs.
The SNMP MIB Reference for information on associated status and alarm conditions.
Verify that the POWER LEDs on the PFUs are green. If they are not, refer to the Checking the LED on the PFU section for troubleshooting information.
 
Packet Processing Card Active LED States
The Active LED on a packet processing card indicates that the software is loaded on the card and that the card is ready for operation. When the system first boots up, all installed packet processing cards are booted into standby mode. The system must then be configured as to which packet processing cards should serve as redundant components (remain in standby mode) and which should function as active components.
The possible states for this LED are described in the following table. If the LED is not green, use the troubleshooting information in the table to diagnose the problem.
Packet Processing Card Active LED States
Verify that the Standby LED on a redundant packet processing card is also blinking green. If so, there is an issue with the card that was active and is transferring its processes.
Refer to the Monitoring the System chapter of this guide for information on determining the status of the packet processing card and system software processes.
Card is not receiving power. OR Card is in Standby Mode.
Verify that the Run/Fail LED is green. If so, the card is receiving power and POST test results are positive. If it is off, please refer to the Packet Processing Card Run/Fail LED States section for troubleshooting information.
Check the state of the Standby LED. If it is green, the card is in standby mode. This is normal operation for the initial power-up. If needed, refer to the Configuring Packet Processing and Line Card Availability section of the Configuring System Settings chapter in this reference for information on making the card active.
 
Packet Processing Card Standby LED States
The Standby LED on a packet processing card indicates that software is loaded on the card, but the card is serving as a redundant component. When the system first boots up, all installed packet processing cards are booted into standby mode. The system must then be configured as to which packet processing cards should be redundant (remain in standby mode) and which should be active.
The possible states for this LED are described in the following table. If the LED is not green, use the troubleshooting information in the table to diagnose the problem.
Packet Processing Card Standby LED States
Verify that the Active LED on the redundant packet processing card is also blinking green.
The Monitoring the System chapter in this guide for show commands, the outputs of which will assist in further determining the problem.
The Configuring and Viewing System Logs chapter in this guide for information on how to view logs.
The SNMP MIB Reference for information on associated status and alarm conditions.
Verify that the Run/Fail LED is green. If so, the card is receiving power and POST test results are positive. If it is off, please refer to the Packet Processing Card Run/Fail LED States section for information on troubleshooting.
Check the state of the Active LED. If it is green, the card is in active mode. If needed, refer to the Manually Initiating a Packet Processing Card Migration section for information on configuring the card to serve as a redundant component.
 
Checking the LEDs on the SPIO
Each SPIO is equipped with the following status LEDs:
In addition to the LEDs listed above, each interface to the management network (both RJ-45 and SFP) are equipped with the following LEDs:
SPIO LED Locations
The possible states for all of the SPIO LEDs are described in the sections that follow.
SPIO Run/Fail LED States
The SPIO Run/Fail LED indicates the overall status of the card. This LED should be green for normal operation.
The possible states for this LED are described in the following table. If the LED is not green, use the troubleshooting information in the table to diagnose the problem.
SPIO Run/Fail LED States
Refer to the Monitoring the System chapter of this guide for information on determining the status of system hardware components.
Verify that the POWER LEDs on the PFUs are green. If they are not, refer to the Checking the LED on the PFU section for troubleshooting information.
 
SPIO Active LED States
The Active LED on the SPIO indicates that the software is loaded on the card and that the card is ready for operation. For the SPIO installed in chassis slot 24, this LED should be green for normal operation. For the SPIO installed in slot 25, this LED should be off for normal operation.
The possible states for this LED are described in the following table. If the LED is not green, use the troubleshooting information in the table to diagnose the problem.
SPIO Active LED States
Card is not receiving power. OR Card in Standby Mode.
Verify that the Run/Fail LED is green. If so, the card is receiving power and POST test results are positive. If it is off, refer to the SPIO Run/Fail LED States section for troubleshooting information.
Check the state of the Standby LED. If it is green, the card is in standby mode. This is normal for the SPIO in slot 25 since the chassis automatically places the card into standby mode at boot up.
SPIO Standby LED States
The Standby LED on the SPIO indicates that software is loaded on the card, but it is serving as a redundant component. For the SPIO installed in slot 25, this LED should be green for normal operation. For the SPIO installed in slot 24, this LED should be off for normal operation.
The possible states for this LED are described in the following table. If the LED is not green, use the troubleshooting information in the table to diagnose the problem.
SPIO Standby LED States
The Monitoring the System chapter in this guide for show commands, the outputs of which will assist in further determining the problem.
The Configuring and Viewing System Logs chapter in this guide for information on how to view logs.
The SNMP MIB Reference for information on associated status and alarm conditions.
Card is not receiving power. OR Card is in Active Mode.
Verify that the Run/Fail LED is green. If so, the card is receiving power and POST test results are positive. If it is off, refer to the SPIO Run/Fail LED States section for troubleshooting information.
Check the state of the Active LED. If it is green, the card is in active mode. This is normal for the SPIO in slot 24 since the chassis automatically makes the card active at boot up.
SPIO Interface Link LED States
The Link LED, associated with a particular SPIO interface indicates the status of the network link. This LED should be green for normal operation.
The possible states for this LED are described in the following table. If the LED is not green, use the troubleshooting information in the table to diagnose the problem.
SPIO Interface – Link LED States
NOTE: This LED will not indicate the presence of a network link until the interface parameters are set during the software configuration process.
Verify that the Run/Fail LED is green. If so, the card is receiving power. If it is off, refer to the SPIO Run/Fail LED States section for troubleshooting information.
SPIO Interface – Activity LED States
The Activity LED associated with a particular SPIO interface indicates the presence of traffic on the network link. This LED should be green when data is being transmitted or received over the interface.
The possible states for this LED are described in the following table. If the LED is not green, use the troubleshooting information in the table to diagnose the problem.
SPIO Interface Activity LED States
 
Checking the LEDs on Ethernet Line Cards
The ASR 5000 can be equipped with a variety of Ethernet line cards that support subscriber traffic. For detailed information about the types of line cards and their applications, refer to the ASR 5000 Installation Guide
The following line cards are currently supported on the ASR 5000:
Each of the Ethernet cards listed above is equipped with status LEDs as listed below:
In addition to the LEDs listed above, each network interface is equipped with the Link and Activity LEDs.
The possible states for all LEDs on these Ethernet line cards are described below.
Ethernet Line Card Run/Fail LED States
The Run/Fail LEDs on the Ethernet line cards indicate the overall status of the cards. These LEDs should be green for normal operation.
The possible states for this LED are described in the following table. If the LED is not green, use the troubleshooting information in the table to diagnose the problem.
Ethernet Line Card Run/Fail LED States
Refer to the Monitoring the System chapter of this guide for information on determining the status of system hardware components.
Verify that the POWER LEDs on the PFUs are green. If they are not, refer to the Checking the LED on the PFU section for troubleshooting information.
 
Ethernet Line Card Active LED States
The Active LEDs on the Ethernet line cards indicate that the operating software is loaded on the card and that the card is ready for operation.
Important: QGLCs and XGLCs only work in an ASR 5000 behind specific types of packet processing cards. Refer to the ASR 5000 Installation Guide for details.
The line cards will remain in a ready mode until their corresponding packet processing card is made active via configuration. While in ready mode the Active LED should be off. After the packet processing card is made active, the line card installed in the upper-rear chassis slot behind the packet processing card will also be made active. The line card (except for the Full-height XGLC) installed in the lower-rear chassis slot behind the packet processing card will enter the standby mode.
The possible states for this LED are described in the following table. If the LED is not green, use the troubleshooting information in the table to diagnose the problem.
Ethernet Line Card Active LED States
Card is in Ready Mode. OR Card is not receiving power. OR Card is in Standby Mode.
This is normal prior to configuration. Neither the Active or the Standby LED on the card is on.
Verify that the Run/Fail LED is green. If so, the card is receiving power and POST test results are positive. If it is off, refer to the Ethernet Line Card Run/Fail LED States section for troubleshooting information.
Check the state of the Standby LED. If it is green, the card is in standby mode. This is normal operation for the initial power-up. If needed, refer to the Configuring Packet Processing and Line Card Availability section of the Configuring System Settings chapter in this guide for information on making the card active.
Ethernet Line Card Standby LED States
The Standby LEDs on the Ethernet line cards indicate that software is loaded on the cards, but are serving as redundant components.
The line cards will remain in a ready mode until their corresponding packet processing card is made active via configuration. While in ready mode, the Active LED should be off. After the packet processing card is made active, the line card installed in the upper-rear chassis slot behind the packet processing card will also be made active. The line card (except for the full-height XGLC) installed in the lower-rear chassis slot behind the packet processing card will also enter the standby mode.
The possible states for this LED are described below. If the LED is not green, use the troubleshooting information in the table to diagnose the problem.
Ethernet Line Card Standby LED States
If green for line cards installed in slots 17 through 23 and 26 through 32, refer to the Monitoring the System chapter of this guide for information on determining the status of the line card and system software processes.
Card is in Ready Mode. OR Card is not receiving power. OR Card is in Active Mode.
Verify that the Run/Fail LED is green. If so, the card is receiving power and POST test results are positive. If it is off, refer to the Ethernet Line Card Run/Fail LED States section for troubleshooting information.
Check the state of the Active LED. If it is green, the card is in standby mode. If needed, refer to the Manually Initiating a Line Card or SPIO Switchover section for information on configuring the card to serve as a redundant component.
 
Ethernet Line Card Interface – Link LED States
The Link LEDs, associated with a particular network interface on the Ethernet line cards, indicate the status of the network link. These LEDs should be green for normal operation.
The possible states for this LED are described in the following table. If the LED is not green, use the troubleshooting information in the table to diagnose the problem.
Ethernet Line Card Interface – Link LED States
NOTE: This LED will not indicate the presence of a network link until the interface parameters are set during the software configuration process.
Verify that the Run/Fail LED is green. If so, the card is receiving power. If it is off, refer to the Ethernet Line Card Run/Fail LED States section for troubleshooting information.
Ethernet Line Card Interface Activity LED States
The Activity LEDs, associated with a particular network interface on the Ethernet line cards, indicate the presence of traffic on the network link. These LEDs should be green when data is being transmitted or received over the interface.
The possible states for this LED are described in the following table. If the LED is not green, use the troubleshooting information in the table to diagnose the problem.
Ethernet Line Card Interface Activity LED States
Checking the LEDs on the RCC
Each RCC is equipped with status LEDs as listed below:
RCC LED Locations
The possible states for all of the RCC LEDs are described in the sections that follow.
RCC Run/Fail LED States
The Run/Fail LED indicates the overall status of the card. This LED should be green for normal operation.
The possible states for this LED are described in the following table. If the LED is not green, use the troubleshooting information in the table to diagnose the problem.
RCC Run/Fail LED States
Refer to the Monitoring the System chapter of this guide for information on determining the status of system hardware components.
Verify that the POWER LEDs on the PFUs are green. If they are not, refer to the Checking the LED on the PFU section for troubleshooting information.
RCC Active LED States
The Active LED on the RCC indicates that the card is being used. For normal operation, this LED should be off on both RCCs.
The possible states for this LED are described in the following table. If the LED is not green, use the troubleshooting information in the table to diagnose the problem.
RCC Active LED States
Refer to the Checking the LEDs on the Packet Processing Cards section to determine which card has failed.
Card is not receiving power. OR Card is in Standby Mode.
Verify that the Run/Fail LED is green. If so, the card is receiving power and POST test results are positive. If it is off, refer to the RCC Run/Fail LED States section for troubleshooting information.
Check the state of the Standby LED. If it is green, the card is in standby mode. This is the normal operating mode.
RCC Standby LED States
The Standby LED on the RCC indicates that software is loaded on the card and is ready to provide a path for data or signalling traffic from a line card to a redundant packet processing card. This LED should be on for normal operation for both RCCs installed.
The possible states for this LED are described in the following table. If the LED is not green, use the troubleshooting information in the table to diagnose the problem.
RCC Standby LED States
Card is not receiving power. OR Card is in Active Mode.
Verify that the Run/Fail LED is green. If so, the card is receiving power and POST test results are positive. If it is off, refer to the RCC Run/Fail LED States section for troubleshooting information.
Check the state of the Active LED. If it is green, the card is in active mode and the RCC is actively routing traffic from a line card installed behind a packet processing card that has failed.
Refer to the Checking the LEDs on the Packet Processing Cards section to determine which packet processing card has failed. Information on determining the cause of the failure can be found in the Monitoring the System chapter of this guide.
 
Testing System Alarm Outputs
The system provides the following two physical alarm mechanisms:
System Audible Alarm: Located on the SMC, the speaker is used to provide an audible indicator that a minor, major, or critical alarm has occurred.
CO Alarms Interface: Located on the SPIO, this interface provides a 10-pin connector that enables three dry-contact relays (Form C) for the triggering of external audio and/or visual indicators. These indicators can be used to alert that either a minor, major, or critical alarm has occurred.
The operation of these alarms can be tested by issuing the following command:
test alarm { audible | central-office [ critical | major | minor ] }
critical: Specifies that the critical CO Alarms output is to be tested.
major: Specifies that the major CO Alarms output is to be tested.
minor: Specifies that the minor CO Alarms output is to be tested.
When this command is executed, the specified alarm is activated for a period of 10 seconds. After this time, the alarm will return to its previous condition.
Taking Corrective Action
In the event that an issue was discovered with an installed application or line card, depending on the severity, it may be necessary to take corrective action.
The system provides several redundancy and fail-over mechanisms to address issues with application and line cards in order to minimize system downtime and data loss. These mechanisms are described in the sections that follow.
Manually Initiating a Management Card Switchover
When the system boots up, the SMC installed in chassis slot 8 will boot into the “active” mode and begin booting other system components. The SMC installed in chassis slot 9 will automatically be booted into “standby” mode dictating that it will serve as a redundant component. However, the active SMC will frequently synchronize currently running tasks or processes with the redundant SMC.
In the event of a critical failure on the SMC in slot 8, system control will be switched to the redundant SMC in slot 9. This is a relatively seamless transition because the two are synchronized. The formerly active SMC will then enter the standby mode allowing it to be safely replaced or restored.
In the event that an issue arises that is not severe enough for the system to perform an automatic switchover, a manual switch over can be invoked by executing the following instructions from the Exec mode prompt:
[local]host_name#
Step 1
card smc switchover
card switch from <24 or 25> to <25 or 24>
You will receive the following prompt:
Are You Sure? [Yes|No]:
Step 2
Press Y to start the switchover.
Step 3
show card table
Check the entry in the Oper State column next to the SMC just switched. Its state should be Standby.
Manually Initiating a Packet Processing Card Migration
When the system boots up, all packet processing cards enter the “standby” mode. The standby mode indicates that the card is available for use but is not configured for operation. Installed components can be made active through the software configuration process. Cards that are not configured to enter the “active” mode will remain in standby mode for use as redundant components.
In the event of the critical failure of a packet processing card, tasks will be automatically be migrated from the active card to a redundant card in standby mode. The line card installed behind the packet processing card that was formerly active will still be used to maintain the interfaces to external network equipment. Installed Redundancy Crossbar Cards (RCCs) will provide a path for signalling and data traffic between the line card and the now active packet processing card. Therefore, redundant packet processing cards do not require that line cards be installed behind them.
In the event that an issue arises that is not severe enough for the system to perform an automatic migration, a manual migration can be invoked. Follow the instructions below to manually initiate a packet processing card migration. These instructions assume you are at the root prompt for the Exec mode:
[local]host_name#
Step 1
card psc migration from <original_slot#> to <final_slot#>
card migrate from <original_slot#> to <final_slot#>
Specifies the chassis slot number of the packet processing card that is being migrated from original_slot which can be any value between 1 through 7, and 10 through 16.
Specifies the chassis slot number of the packet processing card that is being migrated to final_slot which can be any value between 1 through 7, and 10 through 16.
You will receive the following prompt:
Are You Sure? [Yes|No]:
Step 2
Press Y to start the migration.
Step 3
show card table
Check the entry in the Oper State column next to the packet processing card that was just migrated from. Its state should be Standby. The state of the packet processing card migrated to should be Active.
Manually Initiating a Line Card or SPIO Switchover
Ethernet line cards are installed in the half-height slots at the rear of the chassis. This design allows for two half-height line cards to be installed behind every application card (vertical redundancy). With two line cards installed, booting their associated application card causes the card in the upper-rear chassis slot to automatically be made active while the card in the lower-rear chassis slot will automatically be placed in standby mode. In the event that the active card experiences a failure, the system will automatically switch traffic to the standby card in the lower slot.
The XGLC is a full-height card that supports 1:1 side-by-side redundancy. Side-by-side (horizontal) redundancy allows two XGLC cards installed in neighboring slots to act as a redundant pair. Side-by-side pair slots for the XGLC are: 17-18, 19-20, 21-22, 23-26, 27-28, 29-30, and 31-32. If the XGLCs are not configured for side-by-side redundancy, they run independently without redundancy.
When configured for side-by-side redundancy, The XGLC is referenced only by the upper slot number (17 through 23, 26 through 33); the lower slot number is not recognized. All other configuration commands work as if the side-by-side slots were top-bottom slots. Configuration commands directed at the bottom slots either fail with errors or are disallowed.
In the event that a SPIO experiences a failure, the system will automatically switch traffic to the redundant SPIO installed behind the redundant SMC.
In the event that an issue arises that is not severe enough for the system to perform an automatic switchover, a manual switchover can be performed. Follow the instructions below to manually initiate a line card or SPIO switchover. These instructions assume you are at the root prompt for the Exec mode:
[local]host_name#
Step 1
card switch from slot# to slot#
You will receive the following prompt:
Are You Sure? [Yes|No]:
Step 2
Press Y to start the switch.
Step 3
show card table
Check the entry in the Oper State column next to the line card or SPIO that was just switched from. Its state should be Standby. The state of the line card or SPIO switched to should be Active.
Halting Cards
Packet processing cards or line cards that are in either the active or standby modes can be halted. Halting these cards places them into the “offline” mode. In this mode, the card is unusable for session processing as either an active or redundant component.
If a card in the active mode is halted, its tasks, processes, or network connections will be migrated to a redundant component prior to entering the offline mode.
This section describes how to initiate a card halt and restore halted components.
Initiate a Card Halt
Follow the instructions below to manually initiate a card halt. These instructions assume you are at the root prompt for the Exec mode:
[local]host_name#
Step 1
card halt <slot#>
slot# is the chassis slot number in which the card to be halted is installed. It can be any integer value between 1 and 7, 10 through 48. You will receive the following prompt:
Are You Sure? [Yes|No]:
Step 2
Press Y to initiate the halt operation.
Step 3
show card table
Check the entry in the Oper State column next to the line card that was just halted. Its state should be Offline. If the card was in active mode prior to the execution of this command, the state of the redundant component associated with it should now be Active.
Restoring a Previously Halted Card
Follow the instructions below to restore a card that was previously halted. These instructions assume you are at the root prompt for the Exec mode:
[local]host_name#
Step 1
card reboot <slot#> -force
You will receive the following prompt:
Are You Sure? [Yes|No]:
Step 2
Press Y to start the reboot of the card.
Step 3
Verify that the migration was successful by entering the show card table command at the prompt.
Check the entry in the Oper State column next to the line card that was just restored. Its state should be the state of that it was in before it was halted.
Verifying Network Connectivity
There are multiple commands supported by the system to verify and/or troubleshoot network connectivity. Note that network connectivity can only be tested once system interfaces and ports have been configured and bound.
The commands specified in this section should be issued on a context-by-context basis. Contexts act like virtual private networks (VPNs) that operate independently of other contexts. Ports, interfaces, and routes configured in one context cannot be tested from another context without additional configuration.
To switch between contexts enter the following command at the root prompt for the Exec mode:
context <context_name>
context_name is the name of the context to which you wish to switch. The following prompt appears:
[context_name]host_name#
Using the ping Command
The ICMP ping command verifies the system’s ability to communicate with a remote node in the network by passing data packets between and measuring the response. This command is useful in verifying network routing and if a remote node is able to respond at the IP layer. The command has the following syntax:
ping <host_ip_address> [ count <num_packets> ] [ pattern <packet_pattern> ] [ size <octet_count> ] [ src { <src_host_name> | <src_host_ip_address> } ]
<host_ip_address>
host_ip_address specifies the remote node using its IP address entered in IPv4 dotted-decimal format.
count <num_packets>
num_packets must be within the range 1 through 10000. The default is 5.
pattern <packet_pattern>
packet_pattern must be specified in hexadecimal format with a value in the range hexadecimal 0x0000 through 0xFFFF.
packet_pattern must begin with a ‘0x’ followed by up to four hexadecimal digits.
size <octet_count>
octet_count must be a value in the range 40 through 18432. The default is 56.
src { <src_host_name> | <src_host_ip_address> }
src_host_name specifies the source node using the node’s logical host name which must be resolved via DNS lookup.
src_host_ip_address: specifies the source node using its IP address entered in IPv4 dotted-decimal format.
The following displays a sample of a successful response.
PING 192.168.250.1 (192.168.250.1): 56 data bytes
64 bytes from 192.168.250.1: icmp_seq=0 ttl=255 time=0.4 ms
64 bytes from 192.168.250.1: icmp_seq=1 ttl=255 time=0.2 ms
64 bytes from 192.168.250.1: icmp_seq=2 ttl=255 time=0.2 ms
64 bytes from 192.168.250.1: icmp_seq=3 ttl=255 time=0.2 ms
64 bytes from 192.168.250.1: icmp_seq=4 ttl=255 time=0.2 ms
--- 192.168.250.1 ping statistics ---
5 packets transmitted, 5 packets received, 0% packet loss
round-trip min/avg/max = 0.2/0.2/0.4 ms
If no response is received from the target follow these troubleshooting procedures:
If there is still no response, it is likely that the packets are getting discarded by a network device. Use the traceroute and show ip static-route commands discussed to further troubleshoot the issue (see below).
Using the traceroute Command
The traceroute command collects information on the route data will take to a specified host. This is a useful troubleshooting command that can be used to identify the source of significant packet delays or packet loss on the network. This command can also be used to identify bottle necks in the routing of data over the network.
The command has the following syntax:
traceroute { <host_name> | <host_ip_address> } [ count <packets> ] [ df ] [ maxttl <max_ttl> ] [ minttl <min_ttl> ] [ port <port_number> ] [ size <octet_count> ] [ src { <src_host_name> | <src_host_ip_address> } ] [ timeout <seconds> ]
<host_name>
host_name specifies the remote node using the node’s logical host name which must be resolved via DNS lookup.
<host_ip_address>
host_ip_address specifies the remote node using its IP address entered in IPv4 dotted-decimal format.
maxttl <max_ttl>
max_ttl must be specified as a value in the range of 1 through 255. It is an error if max_ttl is less than min_ttl, whether min_ttl is specified or defaulted.
minttl <min_ttl>
min_ttl must be specified as a value in the range of 1 through 255. It is an error if min_ttl is greater than max_ttl, whether max_ttl is specified or defaulted.
port <port_number>
Specifies a specific port to connect to where port_number must be a value in the range 1 through 65535. The default port is 33434.
Specifies the number of bytes each packet. octet_count must be a value in the range 40 through 32768. The default is 40.
src { <src_host_name> | <src_host_ip_address> }
src_host_name: specifies the remote node using the node’s logical host name which must be resolved via DNS lookup.
src_host_ip_address: specifies the remote node using its IP address entered in IPv4 dotted-decimal format.
timeout <seconds>
seconds must be a value in the range 2 through 100. The default is 5.
The following displays a sample output.
traceroute to 192.168.250.1 (192.168.250.1), 30 hops max, 40 byte packets
1 192.168.250.1 (192.168.250.1) 0.446 ms 0.235 ms 0.178 ms
Viewing IP Routes
The system provides a mechanism for viewing route information to a specific node or for an entire context. This information can be used to verify network connectivity and to ensure the efficiency of the network connection. The command has the following syntax:
show ip route [ <route_ip_address> [ <route_gw_address> ] ]
<route_ip_address>
<route_gw_address>
If no keywords are specified, all IP routes within the context’s routing table are displayed.
The following displays a sample of this command’s output showing a context routing table.
"*" indicates the Best or Used route.
Destination Nexthop Protocol Prec Cost Interface
*0.0.0.0/0 10.0.4.1 static 0 0 SPIO1
*10.0.4.0/24 0.0.0.0 kernel 0 0 SPIO1
*10.0.4.0/32 0.0.0.0 kernel 0 0 SPIO1
*10.0.4.3/32 0.0.0.0 kernel 0 0 SPIO1
*10.0.4.255/32 0.0.0.0 kernel 0 0 SPIO1
 
Viewing the Address Resolution Protocol Table
The system provides a mechanism for viewing Address Resolution Protocol (ARP) table information to a specific node or for an entire context. This information can be used to verify that when the system sends an ARP packet, it receives valid responses from other network nodes. The command has the following syntax:
show ip arp [ <arp_ip_address> ]
arp_ip_address specifies a specific network node for which to display ARP information. The address can be entered in IPv4 dotted-decimal or IPv6 colon-separated format. If this keyword is not specified, all entries within the context’s ARP table are displayed.
Important: Restarting the VPN Manager removes all interfaces from the kernel which in turn removes all ARP entries. However, the NPU still retains all of the ARP entries so that there is no traffic disruption. From a user point of view, show ip arp is broken since this command gathers information from the kernel and not the NPU.
The following displays a sample of this command’s output showing a context’s ARP table.
Flags codes:
C - Completed, M - Permanent, P - Published, ! - Not answered
T - has requested trailers
Address Link Type Link Address Flags Mask Interface
10.0.4.240 ether 00:05:47:02:20:20 C SPIO1
10.0.4.7 ether 00:05:47:02:03:36 C SPIO1
10.0.4.1 ether 00:01:30:F2:7F:00 C SPIO1
Using the System Diagnostic Utilities
The system provides protocol monitor and test utilities that are useful when troubleshooting or verifying configurations. The information generated by these utilities can help identify the root cause of a software or network configuration issue.
This section describes how to use these utilities.
Using the Monitor Utility
For troubleshooting purposes, the system provides a protocol monitoring utility. This tool displays protocol information for a particular subscriber session or for every session being processed.
Caution: The monitor tool may cause session processing delays and/or data loss. Therefore, it should be used only when troubleshooting.
Using the Protocol Monitor
The protocol monitor displays information for every session that is currently being processed. Depending on the number of protocols monitored, and the number of sessions in progress, a significant amount of data is generated. You should enable logging on your terminal client to capture all of the information that is generated.
Follow the instructions in this section to invoke and configure the protocol monitoring tool.
Step 1
An output listing all the currently available protocols, each with an assigned number, is displayed.
Step 2
Choose the protocol that you wish to monitor by entering the associated number at the Select: prompt. A right arrow ( > ) appears next to the protocol you selected.
Step 3
Repeat step 2 as needed to choose multiple protocols.
Step 4
Press B to begin the protocol monitor.
WARNING!!! You have selected options that can DISRUPT USER SERVICE
Existing CALLS MAY BE DROPPED and/or new CALLS MAY FAIL!!!
(Under heavy call load, some debugging output may not be displayed)
Proceed? - Select (Y)es or (N)o
Step 5
Enter Y to proceed with the monitor or N to go back to the previous menu.
C - Control Events (ON )
D - Data Events (ON )
E - EventID Info (ON )
I - Inbound Events (ON )
O - Outbound Events (ON )
S - Sender Info (OFF)
T - Timestamps (ON )
X - PDU Hexdump (OFF)
A - PDU Hex/Ascii (OFF)
+/- Verbosity Level ( 1)
L - Limit Context (OFF)
M - Match Newcalls (ON )
R - RADIUS Dict (no-override)
G - GTPP Dict (no-override)
Y - Multi-Call Trace ((OFF))
Q)uit, <ENTER> Display Options, <ESC> Prev Menu, <SPACE> Pause
Step 6
The current state, ON (enabled) or OFF (disabled), is shown to the right of each option.
Step 7
Press the Enter key to refresh the screen and begin monitoring.
The monitor remains active until disabled. To quit the protocol monitor and return to the prompt, press q.
 
Using the Protocol Monitor for a Specific Subscriber
The protocol monitor can be used to display information for a specific subscriber session that is currently being processed. Depending on the number of protocols monitored, and the number of sessions in progress, a significant amount of data is generated. It is highly recommended that logging be enabled on your terminal client in order to capture all of the information that is generated.
Follow the instructions in this section to invoke and configure the protocol monitoring tool for a specific subscriber session.
Step 1
The following screen is displayed, followed by a list of all available monitoring methods:
MONITOR SUBSCRIBER: a) By MSID/IMSI b) By Username c) By Callid d) By IP Address e) By IPv6 Address f) Next-Call g) Next-BCMCS-Call h) Next-BCMCS-Service-Request i) By IMEI j) By MSISDN k) Next-1xRTT Call l) Next-ASNGW Call m) Next-CLOSEDRP Call n) Next-EVDO-Rev0 Call o) Next-EVDO-RevA Call p) Next-GGSN Call r) Next-HA Call s) Next-IPSG Call t) Next-LNS Call u) Next-PDIF Call v) By ASN Peer IP Address w) By PDIF Peer IP Address x) By Peer LAC IP Address y) By SGSN IP Address z) By PCF IP Address 10) By Peer FA IP Address 11) By IPSG Peer IP Address 12) Next-ASNPC Call 13) Next-OpenRP Call 14) Next-rfc3261-proxy Call 15) Next-Proxy-CSCF Call 16) Next-Serving-CSCF Call 17) Next-Interrogating-CSCF Call 18) Next-Proxy-Serving-Interrogating-CSCF Call 19) Next-FNG Call 20) Next-FNG Peer IP Address 21) Next-PHSGW Call 22) By PHS Peer IP Address 23) Next-PHSPC Call 24) Next-Call By APN 25) Next-MME Call 26) Next-SGW Call 27) Next-PGW Call Q) Quit<ESC> Return to Previous Menu
Step 2
Step 3
If no session matching the specified criteria was being processed when the monitor was invoked, the following output is displayed:
NO MATCHING CALL - waiting for a matching call to connect...
C - Control Events (ON ) 11 - PPP (ON ) 21 - L2TP (ON )
D - Data Events (ON ) 12 - A11 (ON ) 22 - L2TPMGR (OFF)
E - EventID Info (ON ) 13 - RADIUS Auth (ON ) 23 - L2TP Data (OFF)
I - Inbound Events (ON ) 14 - RADIUS Acct (ON ) 24 - GTPC (ON )
O - Outbound Events (ON ) 15 - Mobile IPv4 (ON ) 25 - GTPCMGR (OFF)
S - Sender Info (OFF) 16 - A11MGR (OFF) 26 - GTPU (OFF)
T - Timestamps (ON ) 17 - SESSMGR (ON ) 27 - GTPP (ON )
X - PDU Hexdump (OFF) 18 - A10 (OFF) 28 - DHCP (ON )
A - PDU Hex/Ascii (OFF) 19 - User L3 (OFF) 29 - CDR (ON )
+/- Verbosity Level ( 1) 31 - Radius COA (ON ) 30 - DHCPV6 (ON )
L - Limit Context (OFF) 32 - MIP Tunnel (ON ) 53 - SCCP (OFF)
M - Match Newcalls (ON ) 33 - L3 Tunnel (OFF) 54 - TCAP (OFF)
R - RADIUS Dict: (no-override) 34 - CSS Data (OFF) 55 - MAP (ON )
G - GTPP Dict: (no-override) 35 - CSS Signal (OFF) 56 - RANAP (OFF)
Y - Multi-Call Trace (OFF) 36 - EC Diameter (ON ) 57 - GMM (ON )
37 - SIP (IMS) (OFF) 58 - GPRS-NS (OFF)
40 - IPSec IKEv2 (OFF) 59 - BSSGP (OFF)
41 - IPSG RADIUS (ON ) 64 - LLC (OFF)
42 - ROHC (OFF) 65 - SNDCP (OFF)
43 - WiMAX R6 (ON ) 66 - BSSAP+ (OFF)
44 - WiMAX Data (OFF) 67 - SMS (OFF)
45 - SRP (OFF)
46 - BCMCS SERV AUTH (OFF)
47 - RSVP (ON )
48 - Mobile IPv6 (ON )
49 - ASNGWMGR (OFF)
50 - STUN (IMS) (OFF)
75 - App Specific Diameter OFF
 
(Q)uit, <ESC> Prev Menu, <SPACE> Pause, <ENTER> Re-Display Options
Step 4
The current state, ON (enabled) or OFF (disabled), is shown to the right of each option.
Important: Option Y for performing multi-call traces is only supported for use with the GGSN.
Step 5
Repeat step 6 as needed to enable or disable multiple protocols.
Step 6
Press Enter to refresh the screen and begin monitoring.
The following displays a portion of a sample of the monitor’s output for a subscriber named user2@aaa. The default protocols were monitored.
-------------------------------------------------------------------------
Incoming Call:
-------------------------------------------------------------------------
MSID: 0000012345 Callid: 002dc6c2
Username: user2@aaa SessionType: unknown
Status: Active Service Name: xxx1
Src Context: source Dest Context:
-------------------------------------------------------------------------
 
<<<<OUTBOUND 10:02:35:415 Eventid:25001(0)
PPP Tx PDU (9)
PAP 9: Auth-Ack(1), Msg=
 
<<<<OUTBOUND 10:02:35:416 Eventid:25001(0)
PPP Tx PDU (14)
IPCP 14: Conf-Req(1), IP-Addr=192.168.250.70
 
<<<<OUTBOUND 10:02:35:416 Eventid:25001(0)
PPP Tx PDU (27)
CCP 27: Conf-Req(1), MPPC, Stac-LZS, Deflate, MVRCA
 
INBOUND>>>>> 10:02:35:517 Eventid:25000(0)
PPP Rx PDU (30)
IPCP 30: Conf-Req(1), IP-Comp VJ-Comp, IP-Addr=0.0.0.0, Pri-DNS=0.0.0.0,
Sec-DNS=0.0.0.0
 
<<<<OUTBOUND 10:02:35:517 Eventid:25001(0)
PPP Tx PDU (26)
IPCP 26: Conf-Rej(1), IP-Comp VJ-Comp, Pri-DNS=0.0.0.0, Sec-DNS=0.0.0.0
 
INBOUND>>>>> 10:02:35:517 Eventid:25000(0)
PPP Rx PDU (12)
IPCP 12: Conf-Ack(1), IP-Addr=192.168.250.70
 
INBOUND>>>>> 10:02:35:518 Eventid:25000(0)
PPP Rx PDU (31)
LCP 31: Prot-Rej(1), Rejected-Protocol=CCP (0x80fd)
 
INBOUND>>>>> 10:02:35:518 Eventid:25000(0)
PPP Rx PDU (12)
IPCP 12: Conf-Req(2), IP-Addr=0.0.0.0
 
<<<<OUTBOUND 10:02:35:518 Eventid:25001(0)
PPP Tx PDU (14)
IPCP 14: Conf-Nak(2), IP-Addr=192.168.250.87
 
INBOUND>>>>> 10:02:35:519 Eventid:25000(0)
PPP Rx PDU (12)
IPCP 12: Conf-Req(3), IP-Addr=192.168.250.87
The monitor remains active until disabled. To quit the protocol monitor and return to the prompt, press q.
Using the DHCP Testing Tool
The CLI provides a mechanism for testing network connectivity with and configuration of DHCP servers. This functionality can help determine the accuracy of the system’s DHCP configuration and the server’s response time.
This tool provides a mechanism for obtaining an IP address for one or more DHCP servers with which the system communicates.
Important: This tool must be executed from the context in which the DHCP server(s) are configured.
To execute the DHCP test tool enter the following command within the appopriate context:
dhcp test dhcp-service { <service_name> } [ all | <server <ip_addr>|
Sample dhcp test dhcp-service Command Output
 
 

Cisco Systems Inc.
Tel: 408-526-4000
Fax: 408-527-0883